Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities

摘要

Recently, we proposed to transform the outputs of each hidden neuron in amulti-layer perceptron network to have zero output and zero slope on average,and use separate shortcut connections to model the linear dependencies instead.We continue the work by firstly introducing a third transformation to normalizethe scale of the outputs of each hidden neuron, and secondly by analyzing theconnections to second order optimization methods. We show that thetransformations make a simple stochastic gradient behave closer to second-orderoptimization methods and thus speed up learning. This is shown both in theoryand with experiments. The experiments on the third transformation show thatwhile it further increases the speed of learning, it can also hurt performanceby converging to a worse local optimum, where both the inputs and outputs ofmany hidden neurons are close to zero.